Method and apparatus for improving performance on multiple-choice exams

ABSTRACT

A computerized learning approach that enables a user to improve their performance on multiple-choice exams is disclosed. Although the learning approach includes various aspects and implementations, in general, the invention forces test-takers to practice their examination skills and subject matter knowledge in a systematic way. The systematic way forces the users to follow a proven problem-solving approach designed to improve their performance. The invention provides important performance feedback to user&#39;s, for example, elapsed time per question and percentage of correctly answered questions. The invention can also pinpoint for the user the substantive areas of the exam which the user is either weak or strong. Further, the invention is able to assist the user in predicting his/her eventual score.

This is a Continuation application of application Ser. No. 08/315,976,filed Sept. 30, 1994, now U.S. Pat. No. 5,618,182, and the disclosure ofwhich is hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a computerized learning approach, andmore particularly, to a method and apparatus for improving performanceon multiple-choice exams.

2. Description of the Related Art

Multiple-choice examinations are very common today. These examinationstypically test a set of predetermined subject areas and are usually timelimited. One's performance on these exams is very important.

The Multistate Bar Examination (MBE) is a particular multiple-choiceexam for which this invention was developed. The MBE is a rigorousmultiple-choice examination which tests six substantive areas of law ina time limited manner. The MBE forms a significant portion of the barexamination for most of the states in the United States. The ability ofthe test-takers to achieve a passing score on this portion of the barexamination is critical. It is the difference between being able topractice law in a particular state, and not. Consequently, those whodesire to pass the bar examination spend a great deal of time studyingfor the MBE.

Known study approaches make use of written materials and rely on auser's discipline and drive to keep them working. Several bar reviewcourses provide potential examinees with workbooks which provide severalhundred practice questions that the user can work through as he/she seesfit. At the back of these workbooks are contained answers andexplanations for the questions.

A major problem with these known and traditional approaches is that theydo not, and cannot, force the user to study in a consistent, systematicand effective way. As a result, users typically study in a haphazard waywhich varies with their mood, desire and drive. The danger with theseconventional approaches is that user's tend not to develop a consistentproblem-solving approach, but instead develop and utilize inefficientand undesirable study habits. Another serious problem is that users alsotend not to fully understand a question, and why one answer choice iscorrect, while the other answer choices are incorrect.

Thus, there is a need for a system which offers greater efficiency andeffectiveness by requiring the user to study in a consistent andsystematic way.

SUMMARY OF THE INVENTION

Broadly speaking, the invention relates to a computerized learningapproach that enables a user to improve performance on multiple-choiceexams.

A first aspect of the invention concerns a computerized learning methodwhich forces a user to continue attempting to answer a question untilthe user has selected the correct answer choice. This aspect can beimplemented by a computerized learning method which displays a questionand a plurality of answer choices on a display screen, awaits the user'sselection of one of the answer choices, compares the selected answerchoice with a predetermined correct answer choice for the question, andsubsequently awaits selection of another one of the answer choices whenthe selected answer choice is not the correct answer choice.Alternatively, the method can force the user to indicate whether each ofthe answer choices are correct or incorrect.

Another aspect of the invention concerns a computerized learning methodwhich displays an elapsed time for the user to select an answer choicefor the question. This enables the user to monitor and evaluate his/hertime performance on practice questions for the multiple choice exam.This aspect of the invention can be implemented by a computerizedlearning method which displays a question and a plurality of answerchoices on a display screen, enables a timer to monitor a time durationfor the user to answer the question, awaits selection of one of theanswer choices by the user, displays a visual indication of the timeduration, stops the time duration timer when one of the answer choicesis selected, and determines whether the selected answer choice is thecorrect answer choice for the question. Further, a visual indication ofa predetermined time may be displayed together with the visualindication of the time duration.

Yet another aspect of the invention provides the user with assistance sothat the user can better understand the question to be answered or theknowledge needed to answer the question. This aspect can be implementedin one or a combination of the following ways. Typically, however, thisaspect is invoked only when the answer choice selected by the user isnot the correct answer choice. A first implementation provides the userwith a hint towards the correct answer choice. A second implementationdisplays, for the user, substantive information relevant to answeringthe question. A third implementation displays an explanation of thecorrect answer choice once the user has selected the correct answerchoice.

Still another aspect of the invention concerns a computerized learningmethod which provides detailed performance information to the user. Forexample, the invention can plot the time duration for each question orset of questions to produce a graph on a display screen. The graph mayalso include a visual indication of a predetermined time duration.Another example is that the invention can plot the percentage ofcorrectness to produce a graph on the display screen. Here, the graphmay also indicate a visual indication of a target percentage. The canalso predict the user's future performance using the thus far obtainedperformance data on the user.

The invention can also be implemented as an apparatus to improve auser's performance on multiple-choice exams. The apparatus includes acomputer having a display screen associated therewith, and a computerprogram executed by the computer. The computer program includes at leastmeans for displaying a question and a plurality of answer choices on thedisplay screen, means for awaiting selection of one of the answerchoices by the user, means for determining whether the selected answerchoice is the correct answer choice for the question, and means forawaiting selection of another one of the answer choices when theselected answer choice is not the correct answer choice.

Another apparatus implementing the invention concerns a computerdiskette. The computer diskette includes practice questions for amultiple-choice exam, correct answers for the practice questions, and astored computer program for improving performance on the multiple-choiceexam. When the computer program is executed by a computer, the programoperates to carry out any aspects of the methods discussed above. Thecomputer diskette can also include hints for each of the practicequestions, specific topic identifiers for each of the questions; and asubstantive outline containing detailed information on at least one areaof knowledge being tested by the multiple-choice exam.

The invention forces test-takers to practice their examination skillsand subject matter knowledge in a systematic way. The systematic wayforces the users to follow a proven problem-solving approach designed toimprove their performance. Additionally, the invention providesimportant performance feedback to user's, for example, elapsed time perquestion and percentage of correctly answered questions. The inventioncan also pinpoint for the user the substantive areas of the exam whichthe user is either weak or strong. Further, the invention is able toassist the user in predicting his/her eventual score.

The various aspects of the invention discussed above may also becombined in various ways to produce additional implementations of theinvention. In addition, other aspects and advantages of the inventionwill become apparent from the following detailed description, taken inconjunction with the accompanying drawings, illustrating by way ofexample the principals of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be readily understood by the following detaileddescription in conjunction with the accompanying drawings, wherein likereference numerals designate like elements, and in which:

FIG. 1 is a block diagram of an embodiment of an apparatus according tothe invention;

FIGS. 2A and 2B are block diagrams of a first embodiment of a learningmethod according to the invention;

FIG. 3 is a block diagram of a substantive information routine accordingto the invention;

FIG. 4 is a block diagram of a performance evaluation routine accordingto the invention;

FIGS. 5A, 5B and 5C are graphs illustrating a user's performance;

FIG. 6 is a block diagram of a second embodiment of a learning methodaccording to the invention; and

FIGS. 7A and 7B are block diagrams of a third embodiment of a learningmethod according to the invention.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the invention are discussed below with reference to FIGS.1-7. However, those skilled in the art will readily appreciate that thedetailed description given herein with respect to these figures is forexplanatory purposes as the invention extends beyond these limitedembodiments.

FIG. 1 is a block diagram of an embodiment of an apparatus according tothe invention. The apparatus 2 includes a computer 4, a display screen6, an input device 8, and a memory 10. The memory 10 provides storagefor an operating system 12, a learning program 14, practice questions16, user's performance information 18, and miscellaneous information 20.

The computer 4 is preferably a microcomputer, such as a desktop ornotebook computer. However, the computer 4 could also be a largercomputer such as a workstation or mainframe computer. The computer 4could also be remotely located from the user who would interact with thea computer over a network.

The memory 10 is connected to the computer 4. The memory 10 can consistof one or more of various types of data storage devices, includingsemiconductor, diskette and tape. In any case, the memory 10 storesinformation in one or more of the various types of data storage devices.The computer 4 of the apparatus 2 implements the invention by executingthe learning program 14. While executing the learning program 14, thecomputer 4 retrieves the practice questions 16 from the memory 10 anddisplays them to the user on the display screen 6. The user then usesthe input device 8 to select an answer choice for the question beingdisplayed. When the learning program 14 is executed by the computer 4, alearning method according to the invention is carried out. The detailsof various learning methods associated with the learning program 14 aredescribed in detail below in FIGS. 2A, 2B, 3, 4, 6, 7A and 7B.

The learning method according to the invention will cause performanceinformation 18 and miscellaneous information 20 to be produced. Theperformance information 18 may, for example, include a correct/incorrectindicator and an elapsed time for each question or set of questions. Theperformance information 18 may also include a subject and a topic foreach question. The miscellaneous information 20 can include anyadditional data storage as needed by the computer 4, e.g., various flagsand other values which indicate options selected by the user or indicateuser's state of progress. The user's performance information 18 andmiscellaneous information 20 are stored to, or retrieved, from thememory 10 as needed by the computer 4. The operating system 12 is usedby the computer 4 to control basic computer operations. Examples ofoperating systems include Windows, DOS, OS/2 and UNIX.

FIGS. 2A and 2B are block diagrams of a first embodiment of a learningmethod according to the invention. The learning method 22 begins bydisplaying 24 a question and a plurality of answer choices to a user.For example, the question and its answer choices can be retrieved fromthe various practice questions 16 stored in the memory 10 and thendisplayed on the display screen 6. Preferably, the question and itsanswer choices are very similar to the questions and answers whichactually appear on the MBE. It is also preferable that the questions andanswers be displayed in a format and font which are very close to thoseused in the MBE. The closer the appearance and the format of thequestion and its answer to that of the MBE, the more comfortable theuser will be on the actual MBE exam.

Once the question and its answer choices are displayed 24, a questiontimer is started 26. The question timer operates to keep track of theamount of time elapsed from the time the question was displayed untilthe user selects an answer choice. Due to the fact that the MBE is aseverely time limited exam, keeping track of the users time performancefor each question is very important. As the question timer monitors theelapsed time, a visual indication of the elapsed time is displayed 28.For example, a digital stopwatch, a bar graph, or some other graphicaltechnique could be displayed 28 on the display screen 6 to provide avisual indication of the elapsed time to the user. By displaying 28 avisual indication of the elapsed time, the user becomes sensitized tothe amount of time he/she spends to answer questions and how he/she isdoing time-wise with respect to a predetermined duration of time.Alternatively, an audio signal could be used with reduced effectiveness.The visual indication of the elapsed time is far superior to an audiosignal because the user is able to see the elapsed time as he/sheattempts to determine the correct answer choice for the question.

Next, a decision 30 is made based on whether the user has selected ananswer choice for the question. If the user has not yet selected ananswer choice, the learning method 22 awaits the user's selection whileperiodically updating the visual indication of the elapsed time beingdisplayed 28. Once the user has selected an answer choice for thequestion, the question timer is stopped 32. The question timer isstopped at this time so that only the time for the user to selecthis/her first answer choice is measured

A decision 34 is then made based on a comparison of the selected answerchoice and a predetermined answer choice for the question. If theselected answer choice is not the correct answer choice (that is, theselected answer choice does not equal the predetermined answer choice),then the learning method 22 forces the user to keep working on thequestion. Initially, the leaning method 22 displays 36 a hint towardsthe correct answer choice. For example, a hint for the particularquestion could be retrieved from the memory 10 and displayed on thedisplay screen 6 for the user. The hint might identify the issue orstate the appropriate rule of law for the question. Next, the learningprogram 22 again awaits the user's selection 38 of another answerchoice. Preferably, the learning method 22 prevents the user fromselecting an answer choice he/she already incorrectly selected.

Once the user selects 38 another answer choice, a decision 40 is madebased on a comparison of the selected answer choice and thepredetermined answer choice for the question. If the selected answerchoice is still not the correct answer choice, then the learning method22 again forces the user to keep working on the question. However, forthe next go around, the learning method 22 may provide additionalassistance to the user by displaying 42 (or provide the option ofdisplaying) substantive information relevant to the question. Forexample, the substantive information could be a portion of a substantiveoutline of a subject of the MBE. The portion would be the portion of theoutline which discussed the rules of law the user needs to understandand correctly answer the question.

In any case, once the user selects the correct answer choice (afterblock 34 or 40), an explanation of the correct answer choice isdisplayed 44. By displaying such information to the user, the user isencouraged to verify that his/her reasoning for choosing the correctanswer choice was correct, or if his/her reasoning was incorrect, tohelp the user understand the proper approach to the question.

Next, a decision 46 is made based on whether a question set is complete.Although not previously mentioned, the questions are preferablypresented to the user in sets of questions. Preferably, a set couldinclude about fifteen questions. The user is required to work through atleast one entire question set in a single sitting. This forces the userto concentrate on the questions and the problem-solving approach for areasonable period of time (typically 30-60 minutes), even if the userworks through a single set. Using sets of questions also helps tobalance users' performance parameters over the set. Users' performanceparameters tend to be fairly consistent over a reasonable sized set,whereas question by question the parameters tend to have largevariations. In this regard, if the question set is not yet complete, thelearning method 22 will reset the question timer 48 and return to thebeginning of the method 22 to display a next question of the questionset. On the other hand, once the question set is complete, the learningmethod 22 is completed, at least for the given question set.

FIG. 3 is a block diagram of a substantive information routine accordingto the invention. The substantive routine 49 is performed by thecomputer 4 to display 42 substantive information relevant to thequestion as shown in FIG. 2B. The substantive routine 49 begins byidentifying a subject and topic for the question. The topic ispreferably a heading section within a substantive legal outline for aparticular subject of the MBE. For example, the substantive informationfor a question concerning contract law might have topics such asassignment, statue of fraud, acceptance, etc.

After the subject and topic are identified 50 for the question, thecomputer 4 searches 52 the substantive information for the topic. Forexample, the contracts legal outline could be searched for the heading"assignment" using known word searching techniques. Alternatively, thesearching 52 could be performed by a table look-up into a tablecontaining information on the location of topics within the outline. Aportion of the substantive information pertaining to the topic is thendisplayed 54. For example, the portion could be the information in thecontracts legal outline in the section identified by the heading"assignment". Preferably, in a Windows environment, a separate viewingwindow would be opened to contain the portion of the outline, and thequestion and answer choices would be displayed concurrently with theoutline viewing window. It is also preferable that the displayedsubstantive information (e.g., portion of legal outline) have the sameformat and font as the printed outline which the user has available forstudying.

A decision 56 is then made based on whether the question has beenanswered correctly or the user has requested removal of the portion ofthe outline being displayed. Once the substantive routine 49 determines56 that the question has been answered correctly or the user hasrequested removal of the window displaying the portion of the outline,the substantive routine 49 removes 58 the display 54 of the substantiveinformation. Otherwise, the displayed 54 substantive information remainsso that the user can read the pertinent portion of the outline andscroll to other sections if so desired.

FIG. 4 is a block diagram of a performance evaluation routine accordingto the invention. As the user works through the learning method 22,performance information 18 is routinely saved by the computer 4 to thememory 10. At the end of a question set, the performance evaluationroutine 60 would enable a user to display performance information to theuser in a useful way to enable the user to understand his/herperformance. Specifically, the performance evaluation routine 60 beginsby displaying 62 the question number, elapsed time, a correct/incorrectindicator, and subject and topic categories for each question in thequestion set. For example, the displayed information might be displayedin a table such as Table 1 below.

                  TABLE 1    ______________________________________    Question No.              Subject    Topic     Result   Time    ______________________________________    1         Evidence   Hearsay   Correct  3:21    2         Contracts  Assignment                                   Incorrect                                            1:38    3         Contracts  Acceptance                                   Correct  2:20    .         .          .         .    .         .          .         .    .         .          .         .    15        CrimLaw    Battery   Correct  1:58    ______________________________________

Next, a percentage of questions in the question set which were answeredcorrectly is computed and displayed 64. For example, if the user answerseight of the fifteen questions correctly, the percentage displayed wouldbe 53.33%. This percentage is useful to the user because the user candirectly compare his/her set percentage with the percentage the usereventually desires to achieve on the MBE, which is usually at least 65%and typically between 70 and 75%. An average elapsed time for the userto answer the questions in the question set is also be computed anddisplayed 64.

A decision 66 is then made based on whether the user desires toeliminate the correctly answered questions from the master questionsset. Although this decision may be made mandatory to prevent the userfrom repeating questions and thereby polluting his/her performance data,the decision 66 is shown here as being the user's choice. If the userdesires to eliminate the correctly answered questions from the set, thenthe questions answered correctly are disabled 68. This disabling 68 canbe achieved by setting an enable/disable flag associated with each ofthe questions. Such flags are located in the memory 10, e.g., with thepractice questions 16 or the miscellaneous storage 20. Nevertheless, thequestions which have not been answered correctly can be repeated in areview mode, but are preferably not repeated in the practice or studymode, as such would corrupt the user's performance data.

Thereafter, a decision 70 is made based on whether the user desires toview his/her performance history. Here, the computer 4 makes use of theperformance data 18 for each question or set of questions to produceelaborate performance feedback to the user. If the user desires to viewhis/her performance history, the user's performance history is displayed72. Although the performance data could be displayed in tables,preferably, graphical presentations are made. For example, (i) graphs ofusers time verses question set or (ii) correctness verses question setcan be displayed as shown in FIGS. 5A and 5B, respectfully. FIG. 5A is agraph 74 illustrating average elapsed time per question for a set. Atarget elapsed time value 76 is also depicted to provide the user with areference for their desired performance. FIG. 5B is a graph 78illustrating average correctness (as a percentage) for a set. A targetcorrectness percentage 80 is also depicted to provide a reference fortheir desired performance. The graphs 74, 78 are produced by plottingthe average elapsed time and a percentage of correctness for thequestion sets the user has completed. The plotted points can beconnected together with line segments. Alternatively, bar graphs couldbe used.

In any case, these graphs 74, 78 allow the user to see just how his/herperformance is improving. Namely, the user can see the target values foreach performance measure (time, correctness) and how they are fairingand whether their performance is improving, worsening or stable.Although graphs 74, 78 primarily pertain to overall values, similargraphs can also be produced by subject or topic so as to inform the userif certain of the subject areas or topics of the exam are hurtinghis/her overall performance. In fact, the graphs of several subjects ortopics can be simultaneously shown to the user. For example, the averagetime and average correctness for each of the subjects of the exam couldbe simultaneously placed on graphs 74, 78 using different colors orother visually distinguishing marks. Such graphs 74, 78 would alsoinform the user of the user's relative performance by subject or topic.

The invention can also be used to predict the user's performance. Inparticular, the invention can determine and display a user's rate ofperformance improvement, overall or set to set. This rate would providethe user with some indication as to how his/her performance will improvewith future sets. Alternatively, the invention can use the acquiredperformance data 18 on the user to extrapolate out a general trend ofhis/her performance to determine if he/she is on track to meet thegoals. In this regard, a line or curve of best fit for the user could becomputed using known methods and displayed for the user. FIG. 5C is agraph 81 of a curve which uses the user's performance data for sets 1-5to extrapolate out an estimated future performance of the user. Notethat in computing the extrapolated curve it is preferable to use amaximum value for the performance measure. Using a maximum valueprevents the extrapolated curve from being overly biased by inconsistentperformance data, particularly when only a few set of questions havebeen answered, The maximum values are statistical approximations ofusers' maximum performance values, preferably about 75% for correctnessand about 1.5 minutes for elapsed time. From the graph 81 in FIG. 5C,the user will understand that based on his/her performance so far thatto meet the target correctness percentage 80 at least 11 sets will needto be completed.

FIG. 6 is a block diagram of a second embodiment of a learning methodaccording to the invention. The learning method 82 in this embodimentforces the user to indicate whether each of the answer choices iscorrect or incorrect. By forcing the user to consider all the answerchoices, the learning method makes the user practice the problem-solvingapproach employed by most exam takers, namely to make an educated guessat the correct answer after eliminating answer choices known to beincorrect.

In any case, the learning method 82 begins by displaying a question anda plurality of answer choices. Next, the user chooses 86 one of theanswer choices. The learning method 82 then asks the user to indicate 88whether the chosen answer choice is the correct answer choice. Note,here the user can select any of the answer choices, not just the correctanswer choice, and thereafter, indicate whether it is correct orincorrect.

If the user indicates that the selected answer choice is a correctanswer, a decision 90 is made based on whether the selected answerchoice is the correct answer choice. On the other hand, if the userindicates that the selected answer choice is an incorrect answer, adecision 92 is made based on whether the selected answer choice is notthe correct answer choice.

When either decision 90, 92 is answered in the affirmative, then acorrect message is displayed 94 to the user. Alternatively, when eitherdecision 90, 92 is answered in the negative, then an incorrect messageis displayed 96 to the user. Following either block 94 or block 96, thelearning method 82 displays 98 an explanation indicating why theselected answer choice is the correct/incorrect answer choice, therebyallowing the user to confirm the reasoning or analysis behind his/heranswer choice.

Thereafter, a decision 100 is made based on whether all the answerchoices have been selected by the user. If not, blocks 86-98 of thelearning method 82 are repeated until all the answer choices have beenselected, thereby forcing the user to indicate whether each of themultiple answer choices is either correct or incorrect. Once all theanswer choices have been selected, a decision 102 is made based onwhether the question set is complete. If the question set is not yetcompleted, then the learning method 82 returns to block 84 where thenext question and answer choices are displayed for a user according tothe learning method 82. On the other hand, if the question set iscomplete, then the learning method 82 is completed

FIGS. 7A and 7B are block diagrams of a third embodiment of a learningmethod according to the invention. In this embodiment, the learningmethod 104 sequences through the answer choices prompting the user toindicate whether he/she believes the answer choice to be correct orincorrect. The user can also answer "unsure" if the user cannot make aneducated guess at the present time.

The learning method 104 begins by displaying 106 a question and aplurality of answer choices to the user. Next, a selected answer (SA) isset 108 to "A", indicating a first answer choice. The learning method104 then prompts the user to decide 110 whether SA is the correct answerchoice. If the user answers in the affirmative, then block 112 isperformed. Block 112 represents blocks 90 and 94-98 shown in FIG. 6. Onthe other hand, if the user answers negatively, then block 114 isperformed. Block 114 represents blocks 92-98 shown in FIG. 6. A thirdoption is also available to the user. If the user is unsure as towhether or not SA is the correct answer choice, the user can skip theanswer choice.

Thereafter, the learning method 104 performs similar processing for theremaining answer choices. Namely, the selected answer (SA) is set 116,120, 124 to the other answer choices and the user is prompted for adecision 118, 122, 126 just as was done for the first answer choice.Blocks 112, 114 are also used in the same manner for each of the answerchoices.

Once all the answer choices have been processed (answered or skipped),then the same process repeats 128 for the answer choices which the usermay have skipped. The process repeats 128 until the user indicateswhether each answer choice is correct or incorrect. Hence, skippinganswer choices simply delays the decision because the learning methodwill prompt the user for an answer. After all the answer choices havebeen indicated as being correct or incorrect, a decision 130 is madebased on whether the question set is complete. If the question set isnot yet complete, the learning method 104 returns to block 106 toprocess the next question in the same manner. When the question set iseventually completed, the learning method 104 is completed.

Although not shown, the second and third embodiments of the learningmethod (like the first embodiment) can also provide the user with a hintor the option of accessing substantive information to help the useranswer the question. Performance information can be displayed at theusers option.

The above-described embodiments of the learning method can also becombined. A first stage could be designed to focus on the fundamentalsof the proven problem-solving approach. Namely, it may be preferable tostart the user in either the second or third embodiments of the learningmethod because these embodiments stress the basic problem approach.Hence, the second and third embodiments force the user to learn andfollow the desired problem-solving approach. The basic problem-solvingapproach is to read the question, then while reading the answer choices,discarding those answer choices deemed clearly wrong. Thereafter, theremaining answer choices are re-read, and the best answer choice isselected. The question or portions thereof can be re-read as needed.

A second stage could be designed as a practice mode. Here, since theuser would have already become comfortable with the basicproblem-solving approach, the first embodiment of the learning methodwould be used. The first embodiment is particularly useful because itoperates similar to actual exam conditions and offers importantperformance feedback.

Moreover, within the second stage, various levels of study could bepossible. In a first level, all the questions in a set can be for thesame subject. This allows the user to concentrate on questions of thesame subject. This is beneficial because it frees the user from havingto decide which subject category the question pertains to and becausethe user can concentrate on learning the detailed rules pertaining tothe subject. Thereafter, in a second level, the questions in a set canbe from various subjects. Although the second level would be moredifficult than the first level, it would more closely represent theactual exam.

A third level may also be provided. In the third level, the question ina set would again be from mixed subjects but would be more difficultquestions than those in the second level. This level would serve asadvanced level studying for the user.

A fourth level may be provided to permit a user to focus on a specifictopic which he/she wishes to study in depth. For example, if theperformance information indicates that the user is struggling withassignments (topic) in contracts (subject), then the fourth level can beused to practice on question pertaining to assignments. A fifth levelmay be provided to allow the user to repeat questions he/she previouslyanswered incorrectly.

A third stage could be designed as an exam practice mode. In this stagethe user would actually take practice exams under exam-like conditions.The computer system would provide the user with questions, record theuser answers, and time the exam.

The many features and advantages of the invention are apparent from thewritten description, and thus, it is intended by the appended claims tocover all such features and advantages of the invention. Further, sincenumerous modifications and changes will readily occur to those skilledin the art, it is not desired to limit the invention to the exactconstruction and operation as illustrated and described. Hence, allsuitable modifications and equivalents may be resorted to as fallingwithin the scope of the invention.

What is claimed is:
 1. A computerized learning method useful forimproving a user's performance on multiple-choice tests, said methodcomprising:(a) displaying a question and a plurality of answer choiceson a display screen associated with a computer; (b) enabling a timeduration timer to monitor a time duration for the user to answer thequestion; (c) awaiting selection of one of the answer choices by theuser; (d) displaying a visual indication of the time duration; (e)stopping the time duration timer when one of the answer choices isselected; and (f) determining whether the selected answer choice is thecorrect answer choice for the question.
 2. A method as recited in claim1, wherein said method further comprises (g) displaying substantiveinformation relevant to answering the question when requested by theuser.
 3. A method as recited in claim 2, wherein the question has aspecific topic associated therewith, and the substantive information isa portion of an outline which pertains to the specific topic of thequestion.
 4. A method as recited in claim 1, wherein said method furthercomprises:(g) tracking a user's time duration performance and a user'scorrectness performance over a period of time; and (h) predicting theuser's future time duration performance and the user's correctnessperformance based on data obtained from said tracking.
 5. A computerreadable media containing computer instructions that are useful forimproving a user's performance on multiple choice examinations, saidcomputer readable media comprising:first computer program code fordisplaying a practice question and a plurality of answer choices to auser; second computer program code for starting a question timer afterthe practice question and the answer choices are displayed; thirdcomputer program code for receiving the user's selection of one of theanswer choices; fourth computer program code for determining whether theanswer choice selected by the user is the correct answer choice; andfifth computer program code for displaying an explanation of the correctanswer choice for the practice question.
 6. A computer readable media asrecited in claim 5, wherein said computer readable media furthercomprises:sixth computer program code for displaying, at a user'srequest, a portion of an outline that is linked to the particular topicof the practice question that has been presented to the user, theoutline containing detailed information on a plurality of topics in atleast one subject matter area.
 7. A computer readable media as recitedin claim 6, wherein said sixth computer program code displays theportion of the outline that is linked to the particular topic of thepractice question at the user's request, without first having to displaythe explanation of the correct answer choice.
 8. A computer readablemedia as recited in claim 6, wherein after displaying the portion of theoutline that is linked to the particular topic of the practice question,said sixth computer program code allows the user to transition to otheradjacent portions within the outline that pertain to other topics whichare not linked to the practice question.
 9. A computer readable media asrecited in claim 8, wherein said sixth computer program code displaysthe portion of the outline that is linked to the particular topic of thepractice question at the user's request, without first having to displaythe explanation of the correct answer choice.
 10. A computer readablemedia as recited in claim 8, wherein said computer readable mediafurther comprises:seventh computer program code for displaying a visualindication of whether the selected answer choice is the correct answerchoice, the visual indication being displayed immediately andautomatically after the user has selected one of the answer choices andsaid fourth computer program code has determined whether the answerchoice selected by the user is the correct answer choice.
 11. A computerreadable media as recited in claim 26, wherein said computer readablemedia further comprises:seventh computer program code for displaying avisual indication of whether the selected answer choice is the correctanswer choice, the visual indication being displayed immediately andautomatically after the user has selected one of the answer choices andsaid fourth computer program code has determined whether the answerchoice selected by the user is the correct answer choice.
 12. A computerreadable media as recited in claim 6, wherein said computer readablemedia further comprises:seventh computer program code for displaying avisual indication of whether the selected answer choice is the correctanswer choice, the visual indication being displayed immediately andautomatically after the user has selected one of the answer choices andsaid fourth computer program code has determined whether the answerchoice selected by the user is the correct answer choice.
 13. A computerreadable media as recited in claim 12, wherein said computer readablemedia further comprises:eighth computer program code for displaying tothe user a per-question timing indication for the user to answer thepractice question, the timing indication being based on the questiontimer, and the timing indication being displayed at least while the useris answering the practice question.
 14. A computer readable media asrecited in claim 6, wherein said computer readable media furthercomprises:seventh computer program code for displaying to the user aper-question timing indication for the user to answer the practicequestion, the timing indication being based on the question timer, andthe timing indication being displayed at least while the user isanswering the practice question.
 15. A computer readable media asrecited in claim 14, wherein the per-question timing indication beingdisplayed to the user is an elapsed time that the user has spentanswering the question.
 16. A computer readable media as recited inclaim 14, wherein the per-question timing indication is displayed withreference to a predetermined time.
 17. A computer readable media asrecited in claim 16,wherein the question timer does not restrict theamount of time the user has to answer the practice question, wherein thepredetermined time represents a target performance for the user toanswer the practice question, and wherein the per-question timingindication is displayed at least while the user is answering thepractice question and enables the user to evaluate their timeperformance for the practice question with respect to the predeterminedtime.
 18. A computer readable media as recited in claim 6, wherein saidcomputer readable media further comprises:sixth computer program codefor a user to select of a difficulty level for the practice question tobe displayed by said first computer program code.
 19. A computerreadable media as recited in claim 5, wherein said computer readablemedia further comprises:sixth computer program code for monitoring atime duration for the user to answer the question, the time durationbeing based on the question timer; and seventh computer program code fornotifying the user when the time duration exceeds a predetermined time.20. A computer readable media as recited in claim 5, wherein saidcomputer readable media further comprises:sixth computer program codefor displaying to the user a time duration for the user to answer thepractice question, the time duration being based on the question timer,and the time duration being displayed at least while the user isanswering the practice question; and seventh computer program code fornotifying the user when the time duration exceeds a predetermined time.21. A computer readable media as recited in claim 20, wherein thenotifying provided by said seventh computer program code is a visualnotification.
 22. A computer readable media as recited in claim 20,wherein the notifying provided by said seventh computer program code isan audio notification.
 23. A computer readable media as recited in claim5, wherein said computer readable media further comprises:sixth computerprogram code for tracking a user's time and/or correctness performanceover a period of time.
 24. A computer readable media as recited in claim23, wherein said computer readable media further comprises:seventhcomputer program code for predicting the user's future time and/orcorrectness performance based on data obtained from the tracking by saidsixth computer program code.
 25. A computer readable media containingcomputer instructions that are useful for improving a user's performanceon multiple choice examinations, said computer readable mediacomprising:first computer program code for displaying a question and aplurality of answer choices on a display screen associated with acomputer, the question being associated with a topic; second computerprogram code for monitoring a time duration for the user to answer thequestion; third computer program code for detecting selection of one ofthe answer choices by the user; fourth computer program code fordisplaying an indication of the time duration; fifth computer programcode for determining whether the selected answer choice is the correctanswer choice for the question; and sixth computer program code fordisplaying on the display screen an appropriate portion of a substantiveoutline that pertains to the topic of the question without regard to theuser's selection of any of the answer choices, the substantive outlinecontaining detailed information on a plurality of topics.
 26. A computerreadable media as recited in claim 25, wherein the substantive outlineis associated with a multiple choice examination for which the user isattempting to improve their performance, and the substantive outlinecontains at least one subject which includes detailed information on aplurality of topics.
 27. A computer readable media as recited in claim25, wherein said sixth computer program code comprises:computer programcode for identifying the topic of the question; computer program codefor determining the appropriate portion of the outline in accordancewith the topic of the question; computer program code for locating theappropriate portion of the outline to be displayed; and computer programcode for displaying the appropriate portion on the display screen.
 28. Acomputer readable media as recited in claim 25, wherein said computerreadable media further comprises:computer program code for monitoringthe performance of the user in answering a plurality of questions; andcomputer program code for predicting future performance of the userbased on the monitored performance.
 29. A computer readable media asrecited in claim 25, wherein said computer readable media furthercomprises:computer program code for sequentially providing a pluralityof the questions and the answer choices to the user as a question set;computer program code for monitoring the performance of the user inanswering the question set; computer program code for repeating saidsequentially providing and said monitoring for at least another questionset; and computer program code for predicting future performance of theuser based on the monitored performance for the question sets previouslymonitored.
 30. A computer readable media as recited in claim 25,whereinthe appropriate portion of the substantive outline is linked to thetopic of the question, and wherein the displaying of the appropriateportion of the substantive outline on the display screen by said sixthcomputer program code is performed at the user's request.
 31. A computerreadable media as recited in claim 30, wherein during the displaying ofthe appropriate portion of the substantive outline, said sixth computerprogram code operates to allow the user to transition to other adjacentportions within the substantive outline that pertain to other topics.32. A computer readable media as recited in claim 31, wherein saidcomputer readable media further comprises:seventh computer program codefor displaying a visual indication of whether the selected answer choiceis the correct answer choice, the visual indication being displayedimmediately and automatically after the user has selected one of theanswer choices and said fifth computer program code has determinedwhether the answer choice selected by the user is the correct answerchoice.
 33. A computer readable media as recited in claim 30, whereinthe appropriate portion of the substantive outline displayed by saidsixth computer program code has a visual appearance that stronglyresembles that of a corresponding portion of a printed version of thesubstantive outline available to the user.
 34. A computer readable mediaas recited in claim 30, wherein said computer readable media furthercomprises:seventh computer program code for automatically advancing tothe next question after the user selects the selected answer choice forthe current question.
 35. A computer readable media as recited in claim30,wherein the substantive outline is associated with a multiple choiceexamination for which the user is attempting to improve theirperformance, and the substantive outline contains at least one subjectwhich includes detailed information on a plurality of topics, andwherein said computer readable media further comprises:seventh computerprogram code for monitoring the user's performance by tracking averagetime per question and/or percentage correct for each topic; and eighthcomputer program code for displaying performance information for theuser including the average time per question and/or percentage correctfor individual topics and/or subjects associated with the questions. 36.A computer readable media as recited in claim 35, wherein theperformance information displayed is for a most recent session as wellas cumulative of a plurality of sessions.
 37. A computer readable mediaas recited in claim 30, wherein said computer readable media furthercomprises:seventh computer program code for automatically displaying anexplaination of the correct answer choice for the question after theuser has selected one of the answer choices and said fifth computerprogram code determines whether the selected answer choice is thecorrect answer choice.
 38. A computer readable media as recited in claim25, wherein said computer readable media further comprises:seventhcomputer program code for displaying a visual indication of whether theselected answer choice is the correct answer choice, the visualindication being displayed immediately and automatically after the userhas selected one of the answer choices and said fifth computer programcode has determined whether the answer choice selected by the user isthe correct answer choice.
 39. A computer readable media containingcomputer instructions that are useful for improving a user's performanceon a multiple choice examination that covers a plurality of differenttopics, said computer readable media comprising:data storage for aplurality practice questions and corresponding answer choices in anelectronic format, the practice questions and answers beingrepresentative of actual test questions and answers, and each of thepractice questions being associated with a topic; data storage for asubstantive outline in an electronic format, the substantive outlineproviding information on the different topics of the multiple choiceexamination; first computer program code for displaying one of thepractice questions and the corresponding answer choices for the user;second computer program code for monitoring the user's selection of oneof the answer choices; third computer program code for determiningwhether the selected answer choice is the correct answer choice for thequestion; and fourth computer program code for displaying a portion ofthe substantive outline at the user's request independent of whether theselected answer choice is correct or incorrect, wherein the appropriateportion of the substantive outline being displayed pertains to the topicof the practice question displayed.
 40. A computer readable media asrecited in claim 39, wherein the displaying of the appropriate portionof the substantive outline by said fourth computer program codecomprises:computer program code for identifying the topic of thequestion; computer program code for determining the appropriate portionof the outline in accordance with the topic of the question; computerprogram code for locating the appropriate portion of the outline to bedisplayed; and computer program code for displaying the appropriateportion on the display screen.
 41. A computer readable media as recitedin claim 39, wherein the substantive outline contains at least onesubject which includes detailed information on a plurality of topics.42. A computer readable media as recited in claim 39, wherein saidcomputer readable media further comprises:computer program code forsequentially providing a plurality of the questions and the answerchoices to the user as a question set; computer program code formonitoring the performance of the user in answering the question set ofthe questions; computer program code for repeating said sequentiallyproviding and said monitoring for at least another question set; andcomputer program code for predicting future performance of the userbased on the monitored performance for the question sets previouslymonitored.
 43. A computer readable media as recited in claim 39, whereinsaid computer readable media further comprises:fifth computer programcode for monitoring a time duration for the user to answer the practicequestion; and sixth computer program code for displaying a visualindication of the time duration.
 44. A computer readable media asrecited in claim 43, wherein said computer readable media furthercomprises:seventh computer program code for displaying a visualindication of whether the selected answer choice is the correct answerchoice, the visual indication being displayed immediately andautomatically after the user has selected one of the answer choices andsaid third computer program code has determined whether the answerchoice selected by the user is the correct answer choice.
 45. A computerreadable media as recited in claim 39, wherein during the displaying ofthe appropriate portion of the substantive outline, said fourth computerprogram code operates to allow the user to transition to other adjacentportions within the substantive outline that pertain to other topics.46. A computer readable media as recited in claim 45, wherein saidcomputer readable media further comprises:fifth computer program codefor displaying a visual indication of whether the selected answer choiceis the correct answer choice, the visual indication being displayedimmediately and automatically after the user has selected one of theanswer choices and said third computer program code has determinedwhether the answer choice selected by the user is the correct answerchoice.
 47. A computer readable media as recited in claim 46, whereinsaid computer readable media further comprises:sixth computer programcode for monitoring a time duration for the user to answer the practicequestion; and seventh computer program code for displaying a visualindication of the time duration.
 48. A computer readable media asrecited in claim 39, wherein said computer readable media furthercomprises:fifth computer program code for displaying a visual indicationof whether the selected answer choice is the correct answer choice, thevisual indication being displayed immediately and automatically afterthe user has selected one of the answer choices and said third computerprogram code has determined whether the answer choice selected by theuser is the correct answer choice.